In a multimedia streaming service, there are three participants involved: a streaming server, a streaming client and a transmission channel or an underlying network. Usually it is the transmission channel that is the bottleneck of the service, both in terms of throughput and in terms of reliability (i.e., if no throughput bitrate guarantee is assumed), but throughput limitations can occur also at the client and/or at the server.
In a real-time streaming system, due to the dynamically changing throughput characteristics of the channel, client and server, the streaming delivery needs to be adaptive in order to maintain a real-time playback experience for the user. The server should adapt the transmission rate to the varying throughput of the system. An example of such a rate adaptation system can be found in Haskell et al. (U.S. Pat. No. 5,565,924, “Encoder/Decoder Buffer Control for Variable Channel”).
The streaming client provides receiver buffering for storing incoming data before passing them to the media decoder for playout. The receiver buffer is used to compensate for the difference between source encoding rate (also referred to as sampling rate) and transmission rate (pre-decoder buffering). It is also used to compensate for the packet transfer delay variation over the channel (jitter buffering). In general, these two functions are assumed to be combined in a single receiver buffer. However, they can also be implemented with two separate buffers in a receiver, although such an implementation is not optimum from a delay point of view. Receiver buffering can also smooth out the adaptation inaccuracies (i.e. if the system throughput is not matched exactly by the server output).
If the receiver buffer becomes empty (i.e. buffer underflow), which means that the decoder is running out of data to decode, the client needs to pause playout and re-buffer incoming data before resuming. On the other hand, if the incoming data rate is faster than the playout rate, then the receiver buffer space can be exhausted (i.e., buffer overflow), which can result in dropping packets from the buffer in order to make room for new incoming packets. When the packets are dropped, the video quality is degraded. To ensure a smooth and flawless playout, the receiver buffer of the client should be kept within a certain fullness range. In order to guarantee that the receiver buffer will not underflow or overflow, the bitrate for transmission and sampling at the server and that for reception and playout at the client must be adequately controlled.
In the following discussions, bitrate control will be described with reference to the bitrate evolution plots (on the horizontal axis, time in seconds; on the vertical axis, cumulative amount of data in bytes or bits) in which the following curves are defined:                The playout curve, P(t), shows the cumulative amount of data that the decoder has processed by a given time from the receiver buffer;        The sampling curve, S(t), shows the progress of data generation as if the media encoder was run real-time (it is the counterpart of the playout curve, and is actually a time-shifted version of it);        The transmission curve T(t), shows the cumulative amount of data sent out by the server at a given time; and        The reception curve, R(t), shows the cumulative amount of data received and placed into the client buffer at a given time.The difference between any two curves defines the delay and “buffer size” between those two curves. The bitrate control will be constrained by some limits on the difference between two curves (e.g., max buffer size, or max delay). A typical bitrate evolution plot is shown in FIG. 1.        
When determining the best arrangement for server and client cooperation in the bitrate control, the following is to be considered:
A. Sampling curve—the control (i.e., selection of the bitstream for transmission) should be left completely to the server because:
1) it is only the server who knows about the exact characteristics of each bitstream, e.g., switching positions, priority of frames, future frame sizes, and
2) there might not be a bitstream rate that matches the network bit rate, and the server could perform some “tricks” (e.g., thinning, switching up-and-down).
B. Transmission curve—the control (i.e., the rate at which to transmit) should be also left to the server (i.e., using RTCP (Real Time Control Protocol) reports or other bandwidth info from the client) because:
1) in general, it is only the server who can measure the amount of data on-wire, and
2) there might be a need to adapt the transmission rate to the sampling curve if the sampling curve control has limited flexibility.
The server should maintain some real-time constraints by adapting its sampling curve S(t) to its transmission curve T(t). Adaptation of the sampling curve to the transmission curve guarantees that, with adequate buffering, the receiver is able to play out the media with correct synchronization. At every time instant t, the sampling curve S(t) should not deviate from the transmission curve T(t) by too large an amount of bytes.
Mathematically, the constraints maintained by the server at every time instant t is given by−ε≦T(t)−S(t)≦ε′  (R1)where epsilon (ε) and epsilon_prime (ε′) are positive numbers. Epsilon and epsilon_prime directly translate into physical constraints at the receiver. The larger the value “epsilon” (i.e., the more the sampling curve can exceed the transmission curve), the longer the amount of time the client needs to buffer initially in order to guarantee playout without pause. The larger the value “epsilon_prime” (i.e., the more the transmission curve can exceed the sampling curve), the more buffering space a client will need in order to avoid receiver buffer overflow.
If the server operates within a limit as defined by the real-time constraints, the client is responsible to provide any necessary buffering to follow the server within the limit. In that case, the client has to compensate for:
1) pre-decoder buffering for |S(t)−T(t)|, and
2) jitter buffering for transfer delay variation {T(t)−R(t)}.
In addition, the client must tolerate any mismatch between the playout curve and the sampling curve (e.g., clock drift or playback slowdown due to client platform operating system problem).
In a multimedia streaming system, a sender or server needs at each time instant to decide how to encode (at what rate) the following packet it will send and decide at what time to send it. In normal operations, the sender is able to maintain real-time playout by the receiver using only RTCP reports. The 3GPP (3rd Generation Partnership Project) Packet Switched Streaming Service (PSS) defines normative video buffering requirements, which are targeted to compensate for encoding and server-specific delay variations inherent in VBR (Variable Bit Rate) video compression and transmission (see 3GPP TS 26.234 V5.1.0, “Transparent End-to-End Packet Switched Streaming Service (PSS); Protocols and Codecs (Release 5)”, June 2002, hereafter referred to as TS 26.234). When both streaming server and client comply with the buffering requirements, it is guaranteed that the client is able to play out the stream transmitted by the server without client buffer violation (i.e. there will be no buffer underflow or overflow at the client) provided that the stream from the server is transmitted over a constant-delay, reliable transmission channel. However, this is no longer possible under some circumstances such as handovers, retransmissions or clock drift. As a result, the sender and the receiver may take actions conflicting with each other, and a severe degradation in the user experience may occur.
In prior art, a receiver modifies its buffer level by the use of RTSP (Real Time Streaming Protocol) header fields speed (sub-sampling audio and or video) and scale (multiplicative increase or decrease of the bitrate). These headers are defined in IETF RFC2326. The receiver may also make use of bitrate switching commands by the client, as described in “End-to-End bit rate adaptation for PSS”, 3GPP SA4 doc S4-030024.